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ABSTRACT 



An image synthesizing method for synthesizing a plurality 
of images partially overlapping with each other, having the 
steps of inputting a plurality of image data each correspond- 
ing to a plurality of images, extracting a matching point of 
duplicate areas of desired two images partially overlapping 
with each other among a plurality of images of the plurality 
of input image data, and sequentially detecting a position 
relationship between two images in accordance with the 
extracted matching point, detecting the position relation- 
ships of all images in accordance with the position relation- 
ship between two images sequentially detected by the posi- 
tion relationship detecting step, and synthesizing the images 
of the plurality of input image data in accordance with the 
position relationships of all images detected by the all image 
position relationship detecting step to form a synthesized 
image. 
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IMAGE SYNTHESIZING METHOD 

BACKGROUND OF THE INVENTION 

1. Field of the Invention 

The present invention relates to an image synthesizing 
method of synthesizing a plurality of images each including 
a duplicate image area to form a single synthesized image. 

2. Related Background Art 

As a method of synthesizing a plurality of images to form 
an image having a wide angle of view, a method is known 
by which two images are synthesized by designating match- 
ing points in overlapping area of the two images and putting 
one matching point over the other. With this method, the 
matching points of the two images are required to be 
designated by a mouse or the like and this operation is 
cumbersome. 

Extracting matching points may be executed automati- 
cally by template matching or the like to detect the positional 
relationship between two images and synthesize them. 

Such conventional techniques, however, are associated 
with the following problems because a synthesizing process 
for two images is required to be executed a plurality of times 
if a panorama image of 360 degrees is to be formed by 
synthesizing images taken with an ordinary taking lens or if 
a large image with a high resolution is to be synthesized. 

For example, in synthesizing four images, first a synthe- 
sized image of first and second images is formed, and then 
another synthesized image is formed from third and fourth 
images. The two synthesized images are synthesized to form 
a final synthesized image of the four images. 

In this case, image data Ls required to be synthesized after 
the positional relationships between the first and second 
images, between the third and fourth images, and between 
the two synthesized images, are detected. 

It is not so difiBcult to synthesize four images if the 
positional relationships between all images are known in 
advance. However, since a synthesizing process for two 
images is performed a plurality of times to synthesize image 
data, the process eflSciency is poor. 

If the matching points are automatically extracted by 
template matching or the like to detect the positional rela- 
tionship between two images, two images are required to be 
designated at each of a plurality of synthesizing processes 
for two images. Therefore, even if the matching points are 
not required to be manually designated, the operation is 
cumbersome. 

As a method of synthesizing a plurality of images to form 
an image having a wide angle of view, a method is known 
by which two images are synthesized by subjecting the two 
images to geometric conversion such as aflSne transforma- 
tion so as to make the two same points in the duplicate areas 
of the two images coincide with each other. 

With this conventional technique, however, it is necessary 
to recognize the duplicate areas of two images and extract 
the same two points (hereinafter called matching points) 
from the duplicate areas. ITiis operation may be performed 
by displaying two images to be synthesized on a display 
screen and designating the matching points in the two 
images with a mouse or the like. With this method, however, 
the matching points are required to be manually designated 
so that the operation is not simple and some skill is neces- 
sary. 

Template matching is a method of automatically extract- 
ing the matching points firom two images. With this method, 
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one image is divided into small areas and each area is 
compared with the other image to detect the matching points 
which are images most analogous to each other. 

With this template matching, if the duplicate areas of two 

5 images to be synthesized are not designated in advance, 
erroneous matching points may be extracted in some cases 
from the areas other than the duplicate areas of the two 
images. Therefore, an extraction precision of matching 
points is lowered. With a lowered extraction precision of 

30 matching points, two images cannot be synthesized cor- 
rectly. 

Also with this template matching, if a broad duplicate area 
of one image is designated as a search area of a matching 
point and this broad duplicate area contains many analogous 
areas, it takes a long time to extract the matching points from 
the designated two duplicate areas, and therefore the two 
images cannot be synthesized quickly. 

SUMMARY OF THE INVENTION 

20 

It is an object of the present invention to provide an image 
synthesizing method capable of solving the above problems. 

It is another object of the present invention to provide an 
image synthesizing method capable of efficiently synthesiz-^ 

25 ing three or more imafteg with simple operations. 

Under the above objects, according to one embodiment of 
the present invention, an image synthesizing method for 
synthesizing a plurality of images partially overlapping with 
each other is provided which comprises the steps of: input- 

r^n ting a plurality of ima^e data each corresponding to a 
plurality of image , extracting a matching point of duplicate 
areas of desired two images partially overlapping with each 
other among a plurality of images represented by the plu- 
rality of input image data, and sequentially detecting a 

35 position relationship between two images in accordance 
with the extracted matching point; detecting the position 
relationships of all images in accordance with the tx)sitioa 
relationship between two images sequentially detected by 
the position relationship detecting step; and synthesizing the 

40 images represented by the plurality of input image data in 
accordance with the position relationship s of all images 
detected by the all image position relationship detecting step 
to form a synthesized image. 

It is another object of the present invention to provide an 

45 image synthesizing method capable of synthesizing three or 
more images and forming a synthesized image having a 
good image quality at a desired image area, with simple 
operations. 

Under the above objects, according to one embodiment of 

50 the present invention, an image synthesizing method for 
synthesizing a plurality of images partially overlapping with 
each other is provide which comprises the steps of: inputting 
a plurality of image data each corresponding to a plurality of 
images and arranging the plurality of images of the plurality 

55 of input image data in a display area; extracting a matching 
point of duplicate areas of desired two images partially 
overlapping with each other among a plurality of images 
arranged at the arranging step and sequentially detecting a 
position relationship between two images in accordance 

60 with the extracted matching point; forming a data list of an 
order of detecting the position relationship, an image asso- 
ciated with the matching point, and a position of the image 
associated with the matching point in the display area, in 
accordance with the position relationship between two 

65 images in the display area sequentially detected by the 
position relationship detecting step; delecting the position 
relationships of all images from the position relationship 
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between two images sequentially detected, in accordance by the plurality of input image data; setting various param- 
with the data list formed by the data list forming step; and - eters necessary for image conversion synthesization in 
synthesizing the images of the plurality of input image data accordance with the matching point extracted by the match- 
in accordance with the position relationships of all images ing point extracting step; if the various parameters set by the 
detected by the all image position relationship detecting step 5 parameter setting step are for a reduced feature image of the 
to form a synthesized image. reduced feature image data at the matching point extracting 

It is another object of the present invention to provide an step, estimating a duplicate area of a plurality of reduced 

image synthesizing method capable of automatically and ij^ages of the reduced image data having a one-step higher 

precisely synthesizing a plurality of images without com- reduction factor by using the matching point extracted by the 

plicated operations. matching point extracting step and the various parameters 

Under the above objects, according to one embodiment of set by the parameter setting step, setting as a template a 

the present invention, an image synthesizing method for small area for extracting the matching point in each reduced 

synthesizing a pluraUty of images partially overlapping with feature image among the plurality of reduced feature images, 

each other is provided which comprises the steps of: input- and extracting at the matching point extracting step the 

ting a plurality of image data each corresponding to a 15 matching point in the plurality of reduced feature images of 

plurality of images, stepwise reducing the plurality of the reduced feature image data having the one-step higher 

images of the plurality of input image data at different reduction factor, whereas if the various parameters set by the 

reduction factors and generating a plurality of reduced parameter setting step are for an image of the equal-size 

image data of the reduced images; stepwise extracting a feature image data at the matching point extracting step, 

matching point of images of the reduced image data gener- 20 converting at least one image among the plurality of images 

ated by the reduced image data generating step and sequen- of the plurality of input image data to synthesize with 

tially from the reduced image data of a reduced image another image 

having a largest reduction factor to equal-size image data of ^^^^^ ^. j^^^^^j^^ 

an image having the same size as the image represented by become apparent from the following detaUed description of 

the plurahty of input image data; setting various parameters „ embodiments of the invention when read in conjunction with 

necessary for image conversion synthesization m accor- ^^^^ accompanying drawings, 

dance with the matching point extracted by the matching r ^ ^ ct- 

point extracting step; if the various parameters set by the BRIEF DESCRlPTlbN OF THE DRAWINGS 
parameter setting step are for a reduced image of the reduced 

image data at the matching point extracting step, estimating 30 f^IG., 1 is a block diagram showing the structure of an 

a duplicate areaof a plurality of images of the reduced image image synthesizing apparatus according to a first embodi- 

data having a one-step higher reduction factor by using the naent of the invention. 

matching point extracted by the matching point extracting FIG. 2 is an illustrative diagram showing a work window 

step and the various parameters set by the parameter setting with nine images to be synthesized being disposed therein, 

step, setting as a template of a small area for extracting the 35 pjc. 3 is an Olustrative diagram of a process list registered 

matching point in each reduced image among the plurality of in a list storage unit. 

reduced images, and extracting at the matching point ™^ . . n * «• j- r 1 . j 

, . . ' f- I f / J FIG. 4 is an illustrative diagram of a completed process 

extracting step the matching point in the plurality of reduced ° ^ ' 

images of the size reduced image data having the one -step 

higher reduction factor, whereas if the various parameters 40 5 is a flow chart illustrating a procedure of extracting 
set by the parameter setting step are for an image of the ^ matchmg point, the procedure bemg executed by a match- 
equal-size image data at the matching point extracting step, "^8 P°"^' extraction unit. 

converting at least one image among the plurality of images FIG. 6 is a block diagram showing the structure of an 

of the plurality of input image data to synthesize with image synthesizing apparatus according to a second embodi- 

another image. 45 ment of the invention. 

It is a further object of the present invention to provide an FIG. 7 is an illustrative diagram showing a process list 

image synthesizing method capable of automatically and formed in the second embodiment. 

precisely synthesizing a plurality of images in a short FIG. 8 is a block diagram showing the stmcture of an 

processing time. image synthesizing apparatus according to a third embodi- 

Under the above objects, according to one embodiment of so ment of the invention, 

the present invention, an image synthesizing method for piGS. 9A and 9B show examples of input images. 

synOie^zing a plurality of images partially overlapping with j,,^. ^ ^^^^^ illustrating a process algorithm to 

each other is provided which composes the steps ol: input- ^^^^^^^ , 

matching point extraction unit according 

ting a plurality- of image data each corresponding to a ^^^^ embodiment. 

plurality of images, stepwise reducmg, the plurality of 55 ™^ . j. ^ . ^ 

images of the plurality of input image data at different diagram illustratmg a method of cuttmg a 

reduction factors, and generating a plurality of reduced ^^"^P'^^^' ^ executed by the matching point extraction 

image data of the reduced images; extracting a feature image according to the third embodiment. 

of the reduced image of the reduced image data generating P*^- 12 is a diagram illustrating a method of settmg a 

step from the reduced image data generated by the reduced 60 ^^^^^ executed by the matching point extraction 

data generating step and generating reduced feature image according to the third embodiment. 

data; stepwise extracting a matching point of images of FIG. 13 is a diagram illustrating a method of setting a 

feature images of the reduced feature image data generated template cutting area, to be executed by the matching point 

by the feature image data generating step, sequentially from extraction unit according to the third embodiment. 

the reduced feature image data of a reduced image having a 65 FIG. 14 is a diagram illustrating a method of setting a 

largest reduction factor to equal-size feature image data of a search area, to be executed by the matching point extraction 

feature image having the same size as the image represented unit according to the third embodiment. 
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FIG. 15 is a flow chart illustrating a process algorithm to 
be executed by an image conversion synthesization unit 
according to the third embodiment. 

FIG. 16 is a diagram illustrating an image synthesizing 
method according to the third embodiment. 5 

FIG, 17 is a diagram showing an example of a synthesized 
image according to the third embodiment. 

FIG. 18 is a block diagram showing the structure of an 
image synthesizing apparatus according to a fourth embodi- 
ment of the invention. "'^ 

FIG. 19 is a diagram showing a two-dimensional filler 
according to the fourth embodiment. 

FIG. 20 is a flow chart illustrating a process algorithm to 
be executed by the matching point extraction unit according -jj 
to the fourth embodiment. 

FIG. 21 is a diagram illustrating a method of cutting a 
template, to be executed by the matching point extraction 
unit according to the fourth embodiment. 

FIG. 22 is a diagram illustrating a method of cutting a 20 
template, to be executed by the matching point extraction * 
unit according to the fourth embodiment. 

FIG. 23 is a block diagram showing the structure of an 
image synthesizing apparatus according to a fifth embodi- 
ment of the invention. 25 

DETAILED DESCRIPTION OF THE 
PREFERRED EMBODIMENTS 

Embodiments of the image synthesizing apparatus of the 
invention will be described, 

FIG. 1 is a block diagram showing the structure of an 
image synthesizing apparatus of the first embodiment. The 
image synthesizing apparatus is constituted by three parts 
including an interface unit 1, a processing unit 2, and a data 
storage unit 3. Each part is controlled by a control program 
to be executed by a computer. 

The interface unit 1 is constituted by an image display unit 
and an image arrangement unit 11. llie image display unit 10 
displays images designated by a user and a synthesized ^ 
image, on a work window used for image synthesization and 
displayed on a display screen of the computer. The image 
arrangement unit 11 moves a image displayed on the display 
screen by dragging a cursor to arrange the image at a desired 
position. 

The processing unit 2 is constituted by a list formation 
unit 20, a position relationship detection unit 21, and an 
image conversion synthesization unit 22. The list formation 
unit 20 generates a process order and a list of images to be 
processed (hereinafter called a process list). The position 50 
relationship detection unit 21 is constituted by a matching 
point extraction unit 211 and a conversion parameter calcu- 
lation unit 212, calculates the position relationship of two 
images, and outputs conversion parameters. The image 
conversion synthesiz ation unit 22 synthesizes a plurality of 
bodies ^"^^gf, dfllt^ to Obtain one body of ima ge data. 

The data storage unit 3 is constituted by an image data 
storage imit 30, a list storage unit 31, and a conversion 
parameter storage unit 32, respectively for storing image 
data, process data, and conversion parameter. eo 

Next, the operation of the image synthesizing apparatus 
will be described wherein image data of N frames are 
synthesized. When the main program for image synthesiza- 
tion is activated, the work window is displayed on the 
display screen of the computer. At the same time, a file Ust 65 
of image data stored in the image data storage unit 30 is 
displayed in a file management window. 
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A user selects image data to be synthesized from the file 
management window. The image data selected is referred to 
as image #1 hereinafter. The selected image data is read from 
the image data storage unit 30 and the image display unit 10 
displays the image #1 in the work window. This operations 
are repeated N times to display images #1 to #N in the work 
window. 

Next, the user rearranges N images displayed on the 
display screen by using the cursor. At this time, the image 
arrangement unit 11 instructs the image display unit 10 to 
move the displayed images in accordance with the user 
instruction with the cursor The image display unit 10 
displays the images again in accordance with the user 
instruction. At the same time, the image arrangement unit U 
stores arrangement infonmalion of the images in the work 
window rearranged by the user. When the user instructs to 
perform image synthesization by entering a command from 
a keyboard, the processing unit 2 synthesizes the images in 
accordance with the following procedure. 

The list formation unit 20 checks connections between the 
images by using the jnaee #1 as a reference im^y e. in 
accordance with the arrangement information of the image 
data. pG. 2 is an illustrative diagram showing nine images 
to be synthesized , disposed in the work window. Forming 
ttie process list by the list formation unit 20 will be described 
by using nine images as synthesization example. 

The. list formation unit 20 searches a position relationship 
of the images in the work window by tisiog the reference 
jmagfi^ from the left, right, upper, and Tower areas in the" 
work window relative to the reference image. This search is 
performed through comparison with the arrangement infor- 
mation of the reference image and partner image . The 
arrangement information of each image #1 to #9 is repre- 
sented by posl to pos9. The arrangement information con- 
tains vertical and horizontal position data (poslx, posly) to 
(pos9x, pos9y), where (poslx, posly)=(0, 0). The terms "pos 
1" to "pos 9" represent arrangement position data of the 
centers of images #1 through #9 to be synthesized, respec- 
tively. For example, posl to pos9 may be coordinate data of 
the centers of images #1 through #9 in a screen, respectively. 
More specifically, "(poslx, posly)" to "(pos9x, pos9y)" 
represent coordinate positions of the centers of images #1 
through #9 in the horizontal and vertical directions, respec- 
tively. 

The list formation unit 20 compares the arrangement 
information of th e reference image and partner image , and 
searches for the le'tt image, which if it exists, is positioned 
to the left in the horizontal direction, has a threshold value 
thl or larger (i.e., satisfying the conditions (poslx, 
posix)>thl (i=2 to 9) and thl>0 where the right and under 
positions are positive), has a duplicate image area (i.e., 
satisfying the conditions (poslx, posix)<w (i«2 to 9) and 
thl<w, where w is a horizontal width of the image, and has 
a vertical position shift of a minimum value among those 
with a threshold value lh2 or smaller (i.e., |(posly, posiy) 
|<th2 (i=2 to 9). The threshold value thl is about 1/4 of the 
horizontal width w and the threshold value th2 is about 1/4 
of an image vertical width h. In the case of the example 
shown in FIG. 2, no left image of the image #1 exists. 

Similar to the left image search, the right, upper, and 
lower images are searched. For the right image search, the 
arrangement information of t he reference image is compare d 
with the arrangenaent information of the partner image to 
searcD me ngnt linage, wmch is positioned to the ngdt m the 
horizontal direction, has the threshold value thl or larger, 
has a duplicate image area, and has a vertical position shift 
of a minimum value among those with a threshold value th2 
or smaller. 
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For the upper image search, the arrangement information arrangement information of the images #1 and #2, to be the 

of the reference image is compared with the arrangement left half area of the image #2 in the horizontal direction and 

information of the partner image to search the upper image, the +/-10% area from the position of the template on the 

which is at the upper position in the vertical direction, has image #1 in the vertical direction. In this search area, the 
the threshold value lh2 or larger, has a duplicate image area, 5 template is shifted in parallel to calculate a difference 

and has a horizontal position shift of a minimum value between the images #1 and #2. The position where a sum of 

among those with a threshold value thl or smaller. absolute values of differences is minimum, is determined as 

„ , . . . • f *• ttie position of a matching point (step S14). 

For the lower image search, the arrangement mformation n ,• u-i * r*u if ^ • . o^>4 • i_ i j 

c .X. c • ■ J Tl . Reuability of the results obtamed in step S14 is checked 

of the reference unage is compared with he arrangement ^ 5^5) ^ ^ ^^^^^^^^ ^^^^^^^ 

m ormaiion or tne partner .mage to searcn tne ower .mage, - ^^-^^ completed for aU the cut templates (steD 

which IS at the ower position in the vertical direction, has s^g) jf completed, the flow returns to step S13, whereat 

the threshold value lh2 or larger, has a duplicate image area, completed, the process is terminated, 

and has a horizontal position shift of a minimum value judgment of the reliabiHty in step S15 is performed by 

among those with a threshold value thl or smaller. y^ing niinimum sum of the absolute values of differences 

Therefore, the image #2 is selected as the right image oL and the second smallest sum. In this embodiment, the 

the'image #1, the image #4 is selected as the upper image, matching point is judged reliable if the minimum sum of the 

and the image #7 is selected as the lower image. The absolute values of differences has a first predetermined 

selection results are stored in the list storage unit 31. FIG. 3 threshold value th3 or smaller and a difference between the 

is an illustrative diagram of the process list registered in the minimum sum and the second smallest sum of the absolute 
list storage unit 31. The list formation unit 20 judges at this ^° values of differences is a second threshold value th4 or 

stage whether all the images have been registered in the larger. If it is judged reliable, the coordinates of the matching 

process list. If all are registered, the process is terminated. If points in the images #1 and #2 are stored in a memory of the 

not, with reference to the images registered in the process matching point extraction unit 211. In this embodiment, the 

list, the similar process using the reference image is per- position where the sum of the absolute values of differences 

formed for the images still not registered in the process list, is minimum, is used as the position of the matching point, 

to thereby complete the process list. In this example, there The matching point position may alternatively be the posi- 

are remaining images, and so the process continues. lion having the maximum correlation value obtained by a 

With this continued process, the image #3 is selected as correlation calculation, 

the right image of the image #2, the image #5 is selected as 1° above description, the matching point extraction 

the upper image, and the image #8 is selected as the lower unit 211 extracts the matching points of the image #1 and the 

image. This selection results are stored in the list storage unit "ght image #2. In other cases, for example, in extracting the 

31. In this case, right, left, upper, and lower images for the matching points of the image #1 and upper image #4, it is 

images #3 and #7 cannot be found when they are searched confirmed from the process list that the image #4 is at 

for. Lastly, the image #6 is selected as the upper image of the l^e upper position of the image #1. At step Sll, the template 

image #3, and the image #9 is selected as the right image of cutting area is set to the upper portion of the image #1, and 

the image #8. The selection results are stored in the list ^tep S13 the search area is set to be the lower half area of 

storage unit 31. After the list formation unit 20 confirms that l^e image #4 in the vertical direction and the +/-10% area 

all the images have been registered in the process list, it the position of the template on the image #1 in the 

terminates the process. FIG. 4 is an illustrative view of the horizontal direction. 

completed process list. Similarly, for the left and lower images, the matching 

After the process list is formed, the position relationship PP^"^^ extracted by setting the template cutting area and 

detection unit 21 sequentiaUy obtains the position relation- ^^^^^ ^^^^ i" accordance with the arrangement information, 

ship of two images in accordance with the completed Next, the conversion parameter calculation unit 212 esti- 
process list. Specifically, with reference to the process list 45 n^ates coordinate transformation parameters to be used for 

shown in FIG. 4, the position relationship is obtained in the io^age synthesization in accordance with the extracted 

order of images stored in the process ILst, i.e., in the matching point positions. Affine transformation is used for 

sequential order of the images #1 and #2, images #1 and #4, coordinate transformation. Assuming that the image #2 has 

images #1 and #7, images #2 and #3, images #2 and #5, a position relationship, relative to the image #1, of 6 
images #2 and #8, images #3 and #6, and images #8 and #9. 50 rotation, (dx, dy) parallel motion, and m magnification, a 

The operations of the matching point extraction unit 211 P^^"* (^1' i° ^"^^g^ corresponds to a point (x2, y2) 

and conversion parameter extraction unit 212 of the position indicated in the following equations (1). 
relationship detection unit 21 will be described by taking as 

an example the position relationship of the images #1 and x2 = [co^'xJ ^swO^yi -dx)xm (i) 

#2. FIG. 5 is a flow chart illustrating the procedure of 55 ^^A xi + B yi -t-c 
extracting a matching point, to be executed by the matching 

point extraction unit 211. First, a template cutting area of the {-^nO-xi + cos9-yi -dy)xm 

image #1 is set (step Sll). The right side area is set as an =-B x1 -i-A yi -^D 
approximate template cutting area with reference to the 
arrangement information of the images #1 and #2, and each go 

template is cut as a small block area equal to about 10% of where A=m cos 6, B=m sin 6, C=-m dx, and Do-m dy. 

the image size. Xhe conversion parameter calculation unit 212 estimates 

For all cut templates, the following steps S13 to S15 are the conversion parameters A, B, C, and D by a least square 

executed (step S12). Specifically, a search area for searching method. The obtained conversion parameters A, B, C, and D 
a point matching each cut template is set for the image #2 65 are stored in the conversion parameter storage unit 32. 

(step 813). In this case, since the duplicate area cannot be Similar processes are sequentially performed for pairs of 

identified in advance, the search area is determined, from the other images with reference to the process list to obtain 
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position relationships. After all the position relationships for the display screen by the image display unit 10. Image 
the images registered in the process list are obtained, the synthesization is not limited only to nine images, but gen- 
image conversion synthesization unit 22 executes an image erally N images can be synthesized with simple operations 
synthesization process in accordance with the obtained as described above. 

conversion parameters. 5 FIG. 6 is a block diagram showing the structure of the 

In accordance with the conversion parameters stored in ^^^^ synthesizing apparatus of the second embodiment. In 

the conversion parameter storage unit 32 for each image in pjc. 6, like constituent elements to those of the first embodi- 

the process list^^lhe image conversion synthesization unit 22 ^^^^ represented by using identical reference numerals, 

sets area of the synthesized image for the referen^ I" the second embodiment, as different from the first 

#1. The synthesized unage area is represented by the right, . , • l r 

left, upper, and lower corners thereof In Ihe example shown '° embodiment the processing umt 2 has a reference miage 

in HG. 2. the left end position is obtained as the leftmost ^^""'e ^3 and a position relationship synthesization umt 

position pixel from the reference image #1 among the six ^ 

pixels including the upper left comer pixels and lower left reference image setting unit 23 searches an image 

corner pixels of the images #1, #4, and #7. For example, the positioned at a generally central area of the work window in 

position of the upper left comer of the image #7 relative to accordance with the arrangement information to thereby set 

the reference image #1 is obtained through inversion trans- the reference image. The position relationship synthesiza- 

formation of affine transformation as in the following equa- lion unit 24 obtains a position relationship of each image 

tions (2), by using composite parameters A17, B17, C17, relative to the reference image and synthesizes the position 

and D17 of the images #1 and #7. A point (xLl, yLl) is the relationships, in accordance with the position relationship of 

coordinates of the upper left corner point (xL7, yL7) of the 20 images stored in the conversion parameter storage unit 32. 

image #7 as referenced to the reference image #1. In the following, synthesization of nine images shown in 

FIG. 2 will be described. 

xLi'^iTxLl+Bi7yL7+ciT When a user arranges the nine images at the positions 

yLl'-BiT xLl+AiryLl+DiT (2) shown in FIG. 2, the reference image setting unit 23 obtains 

25 first the minimum and maximum values of position data in 

where A17'«A17/(A17^+B17^), B17'=-B17/(A17^+ the horizontal and vertical directions from the arrangement 

B17^), Cir=(-A17'C17+B17 D17)/(A17^+B17^). and information of each image in the work window, and aver- 

D17'«(-B17 C17-A17 D17)/(A17^+B17^). ages the minimum and maximum values to obtain the 

Similarly, the right end position is obtained as the right- median values of the position data in the horizontal and 

most position pixel from the reference image #1 among the 30 vertical directions. The image having the arrangement infor- 

six pixels including the upper right comer pixels and lower mation with the shortest distance from the median values is 

right corner pixels of the images #6, #3, and #9. The upper set as the reference image. The image set as the reference 

end position is obtained as the most upper pixel from the image positions at a generally central area of the work 

reference image #1 among the six pixels including the upper window so that the image #2 is set as the reference image, 

left comer pixels and upper right comer pixels of the images 35 The list formation unit 20 performs processes similar to 

#4, #5, and #6. The lower end position is obtained as the the first embodiment to form the process list. FIG. 7 is an 

most lower pixel from the reference image #1 among the six illustrative diagram showing the process list formed by the 

pixels including the lower left corner pixels and lower right second embodiment. After the process list is formed, the 

corner pixels of the images #7, #8, and #9, If, for example, position relationship detection unit 21 sequentially obtains 

the position of the lower right comer pixel of the image #9 40 the position relationships of pairs of two images in accor- 

as referenced to the reference image #1 is to be obtained, the dance with the formed process list. 

conversions by the equations (2) are executed twice because The position relationship synthesization unit 24 sequen- 

the direct conversion parameters are not stored in the tially obtains the position relationships of images relative to 

conversion parameter storage unit 32. the reference image in accordance with the position rela- 

After the rectangular area for the synthesized image is 45 tionship of images stored in the conversion parameter stor- 

obtained by the above processes, the image conversion age unit 32, and synthesizes the position relationships, 

synthesization unit 22 obtains each pixel value from the It can be seen from the process list shown in FIG. 7 that 

image data of the images #1 to #9. The following processes the position relationship detection unit 21 has already 

are performed for each pixel. detected the position relationships of the images #1, #3, #5, 

Each pixel position (referenced to the reference image #1) 50 #8 relative to the image #2 and the conversion parameters 

for the synthesized image is converted into the pixel position have already been stored in the conversion parameter stor- 

of the images #2 to #9 by using the conversion parameters age unit 32. The position relationships of the images #4, #7, 

stored in the conversion parameter storage unit 32. For #6, and #9 are related to the images #1, #1, #3, and #8.Tl is 

example, the pixel position of the image #2 is obtained therefore necessary to synthesize the conversion parameters 

through conversion of the equations (1). The pixel position 55 in order to obtain the position relationships relative to the 

of the image #3 is obtained by executing similar conversion reference image #2. 

twice by using the conversion parameters of the images #2 In obtaining the position relationship of the image #4 

and #3 and the images #1 and #2. relative to the reference image #2, the position relationship 

The pixel values of the obtained pixel positions are synthesization unit 24 first reads from the conversion param- 

averaged to obtain a pixel value for the synthesized image. 60 eter storage unit 32 the conversion parameters A21, B21, 

In this case, the pixel values at the pixel position after C21, and D21 representative of the position relationship 

conversion is in the area outside of the original image area, between the images #2 and #1 and the conversion param- 

the pixel values are not averaged. A pixel which is not eters A14, B14, C14, and D14 representative of the position 

contained in any of the image areas of the images #1 to #9 relationship between the images #1 and #4. The conversion 

is assigned a dummy pixel (e.g., white pixel). 65 parameters A24, B24, C24, and D24 representative of the 

. I^e image data synthesized by the above processes is position relationship between the images #2 and #4 are 

written in the image data storage unit 30 and displayed on obtained by the following equations (3). 
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A2M14A2UB14B21 
B2Ml4B2l+BU'A2l 
C2MiAC2i+BUD2i+CU 

514C21+A14/)21+£H4 (3) 

The obtained conversion paranaeters A24, B24, C24, and 
D24 are overwritten in the conversion parameter storage unit 
32 at the storage area for the conversion parameters A14, 
B14, C14, and D14. Similarly, the position relationships of 
the images #7, #6, and #9 relative to the reference image #2 
are calculated and overwritten in the conversion parameter 
storage unit 32. 

After the position relationships of the images registered in 
the process list relative to the reference image #2 are 
synthesized, the image conversion synthesization unit 22 
performs an image synthesization process in accordance 
with the synthesized conversion parameters. First, in accor- 
dance with the conversion parameters stored in the conver- 
sion parameter storage unit 32 for each image registered in 
the process list, the image conversion synthesization unit 22 
sets an area for the synthesized image as referenced to the 
reference image #2. After the rectangular area for the 
synthesized image is determined, the image conversion 
synthesization unit 22 obtains a pixel value of each pixel of 
the image data of the images #1 to #9, The conversion of the 
pixel position of each pixel as referenced to the pixel 
position of the reference image is completed only by one 
conversion because the position relationships of the images 
have already been synthesized by the position relationship 
synthesization unit 24. 

With the image synthesizing apparatus of this 
embodiment, the image conversion synthesization unit 22 
can obtain the pixel position as referenced to the reference 
image only by one conversion, even for the image area such 
as image #4 which has no direct right, left, upper, or lower 
relationship to the reference image #2. Therefore, the con- 
version efficiency of the pixel position by the image con- 
version synthesization unit 22 is good. 

As compared to the synthesized image by the image 
synthesizing apparatus of this embodiment, the synthesized 
image by the image synthesizing apparatus of the first 
embodiment has the best image quality at the area of the 
image #1 because the image #1 is used as the reference 
image and this area is not substantially subjected to the 
image conversion. In contrast, the synthesized image by the 
image synthesizing apparatus of this embodiment has the 
best image quality at the area of the image #2 because the 
image #2 is used as the reference image and this area is not 
substantially subjected to the image conversion. 

In this embodiment, the reference image is set in accor- 
dance with the arrangement information of images set by the 
user. The reference image may be set in accordance with the 
position relationship obtained by the position relationship 
detection unit 21. Furthermore, in this embodiment, the 
reference image is set so as to make the image quality at the 
generally central area of the synthesized image be set best. 
The area of the best image quality may be set by using a 
cursor and the image in this area is set as the reference 
image. 

The present invention may be applied to a system con- 
stituted by a plurality of apparatuses or to a system consti- 
tuted by a single apparatus. Obviously, the scope of the 
invention includes a system or apparatus which realize the 
functions of the invention by using a program supplied 
thereto. In this case, the effects of the invention can be 
obtained if the system or apparatus reads a storage medium 
storing software programs realizing the functions of the 
invention. 
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According to the image synthesizing apparatus of the 
embodiments, three or more images can be efficiently syn- 
thesized to form an image with a broad angle of view. 
Further, three or more images can be synthesized with 
5 simple operations to form an image with a broad angle of 
view. Furthermore, it is possible to efficiently synthesize 
images and obtain a synthesized image having a good image 
quality at a desired image area. 

FIG. 8 shows the structure of an image synthesizing 
10 apparatus according to the third embodiment of the inven- 
tion. 

In FIG. 8, image data a and image data b to be synthesized 
are applied to input terminals 100a and 100/?. The image 
data a and image data b are obtained by an unrepresented 

IS electronic still camera, video camera, or the like and have a 
partial duplicate image area. For example, the image data a 
corresponds to an image shown in FIG. 9 A, and the image 
data b corresponds to an image shown in FIG. 9B. 

Reference numerals 111 and 112 represent an image 

20 reduction unit which reduces the size of the image data a and 
b, for example, by 1/2, 1/4, and 1/8 and hierarchically 
generates the reduced image data which is stored in image 
memories of the image reduction units 111 and 112. 
Reference numeral 20 represents a matching point extrac- 

25 tion unit which extracts the matching points of the images 
represented by the image data a and b supplied from the 
input terminals lOOfl and WOh or by the reduced image data 
generated by the image reduction units 111 and 112. 
Reference numeral 130 represents a matching point vector 

30 selection unit which selects and outputs only the matching 
point vector having high reliability among matching point 
vectors of the matching points extracted by the matching 
point vector extraction unit 120. 

Reference numeral 140 represents a parameter estimation 

35 unit which estimates coordinate transformation parameters 
for image coordinate transformation in accordance with the 
matching point vector selected by the matching point vector 
selection unit 130. 

Reference numeral 150 represents an image conversion 

40 synthesization unit which converts the coordinates of the 
input image data a and b in accordance with the coordinate 
transformation parameters estimated by the parameter esti- 
mation unit 140, and synthesizes the converted image data 
to output synthesized image data c to an output terminal 160. 

45 The whole of the image synthesizing apparatus is con- 
trolled by a control unit not shown. 

The operation of the image synthesizing apparatus of the 
third embodiment will be described wherein the synthesized 
image data c is formed by the input image data a and b. 

50 In this embodiment, the input image data a and b are each 
having 512x512 pixels providing a gradation image. 

llie image reduction units 111 and 112 reduce the size of 
the image data a and b, for example, by 1/2, 1/4, and 1/8 and 
generates the reduced image data of 256x256 pixels, 128x 

55 128 pixels, and 64x64 pixels, the reduced image data being 
represented hereinafter by a2 and b2, a4 and b4, and a8 and 
b8. 

The matching point extraction unit 120 first extracts the 
matching points of the reduced image data a8 and b8 of the 

60 input image data a and b reduced by 1/8, and then sequen- 
tially extracts the matching points of the reduced data having 
a reduction factor of 1/4 and 1/2 and of the equal size of the 
input image data. In extracting the matching points of the 
reduced data having a reduction factor of 1/4 and 1/2 and of 

65 the equal size of the input image data, estimation at the 
duphcate area is performed by using the conversion param- 
eters formed by the matching points extracted previously. 
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The matching point extraction algorithm to be executed b8 set as above is compared with the image in the template 

by the matching point extraction unit 120 will be described area for the left side reduced image data a8 while sequen- 

with reference to the flow chart shown in FIG. 10. tially shifting the position of the template. Adifference pixel 

Referring to FIG. 10, at step S21 a template cutting area , value between pixels constituting the image in the template 

for the reduced data a8 is set. In extracting the matching 5 and the image in the search area is calculated, and the 

points of the reduced data a8 and b8, since the two duplicate position of the template where the sum of absolute values of 

areas of the two reduced images cannot be known in the difference values becomes minimum is delected as the 

advance, a predetermmed area is set as the template cuttmg matching point 

'Tnthisembodiment,assumingthattheinputimagesofthe , In this embodiment, the position where th^ 

input image data a and b arc arranged to the left and right as ^f^;^^^^^ of differences is mmimum is used as the position 

shown in FIGS. 9 A and 9B, the template cutting area (an °5 ^^^^^^^mg pomt. The matchmg point position may be 

area T in FIG. 11) of the left side reduced image data a8 is P^^^^i^" ^^^^"S maximum correlation value obtamed 

determined to be the 30% area of the image on the right side correlation calculation between the images in the tem- 

in the horizontal direction and the 10% to 90% area of the P^^*^ ^"^ ^" ^^e search area. 

image in the vertical direction. ^5 At step S25 shown in FIG. 10, reliability of the matching 

Next, at step S22 of FIG. 10, blocks are cut from the poi°t detected at step S24 is checked. The reliability check 

template cutting area set at step S21 of FIG. 10 as templates. is performed by using the template position where the sum 

Each block has an area about 10% of the original image size of absolute values of differences is minimum and the tem- 

■as indicated by broken lines in FIG. 11 and has about 7x7 plate position where the sum of absolute values of differ- 

pixels. 20 ences is second minimum. 

If the input images of the input image data a and b are Specifically in this embodiment, it is judged that the 

arranged at the upper and lower positions, the template detected matching point is reliable if the minimum sum of. 

cutting area of the upper side reduced image data a8 is absolute values of differences has a predetermined threshold 

determined to be the 30% area of the image on the lower side value or smaller and if the second minimum value of 

in the vertical direction and the 10% to 90% area of the 25 absolute values of differences has another predetermined 

image in the horizontal direction. threshold value or larger. The coordinate data of the match- 

The following steps S23 to S25 are executed for all the ing point extracted from the images of the reduced image 

templates cut at step S22 of FIG. 10. In step S26, it is data a8 and b8 is stored in' an internal memory of the 

discriminated whether or not all of the plurality of images matching point extraction unit 120. 

have undergone the synthesizing processing. More 30 The processes of steps S23 to S25 are executed for each 

specifically, in this step, the number of images that have template cut at step S21 until it is detected that the processes 

undergone the foregoing steps is counted and the synthesiz- are completed for all the templates, 

ing processing is ended when the count value is equal to the Next, the process to be executed by the matching point 

number of images (for example, nine). vector selection unit 130 will be described. 

At step S23 of FIG. 10, for each template cut in the above 35 The matching point vector selection unit 130 first reads 

manner, a search area for searching the matching point in the coordinate data of each matching point extracted from the 

image of the right side reduced image data b8 is set. In images represented by the reduced image data a8 and b8 

extracting the matching points of the reduced image data a8 extracted by the matching point extraction unit 120 and 

and b8, the duplicate areas of the two images cannot be stored in the internal memory, and calculates the matching 

known in advance so that a predetermined area is set as the 40 point vector through calculation of read coordinate data in 

search area for the matching point. the horizontal and vertical directions. Histograms of all the 

In this embodiment, the search area for the matching point calculated matching point vectors are generated both in the 

of the right side reduced image data b8 corresponding to horizontal and vertical directions. From the generated his- 

each template of the plurality of templates cut for the left tograms of matching point vectors both in the horizontal and 

side reduced image data a8 is set to the area including the 45 vertical directions, only the matching point vectors in the 

template area and the area shifted about a half area to the highest occurrence frequency range are selected and output 

right from the right side end of the template in the horizontal to the next stage parameter estimation unit 140. 

direction and the area including the template area and the As above, the matching point selection unit 130 selects 

area about 10% upper and lower than the template area in the only the matching point vectors in the highest occurrence 

vertical direction. 50 frequency range. Therefore, even if the matching point 

FIG. 12 shows an example of the search area of the extraction unit 120 extracts erroneous matching point 

matching point for the right side reduced image data b8. In vectors, only the matching point vector data with high 

FIG. 12, the search area of the matching point for the right reliability can be output to the parameter estimation unit 

side reduced image data b8 corresponding to the template of 140. 

the left side reduced image data a8 indicated by oblique lines 55 The parameter estimation unit 140 estimates coordinate 

is indicated at S. conversion parameters in accordance with the matching 

Setting the search area of the matching point is based on point coordinates of the reduced image data a8 and b8 

the assumption that the duplicate areas of the image repre- represented by the matching point vector data supplied from 

sented by the input image data a and b are not displaced by the matching point vector selection unit 130. 

50% or more of the input image size in the horizontal 60 Estimation of the coordinate conversion parameters will 

direction and by upper and lower 10% areas of the input be later detailed. 

image size in the vertical direction. If this assumption of the The matching point vector data for the size reduced image 

duplicate areas represented by the input image data a and b data a8 and b8 with a reduction factor of 1/8 obtained as 

are not satisfactory, the initial setting for the search area of above is used for estimating the duplicate areas of two 

the matching point is changed. 65 reduced images at the next matching point extraction pro- 

At step S24 shown in FIG. 10, the image in the search area cess for the reduced image data a4 and b4 with a reduction 

of the matching point for the right side reduced image data factor of 1/4. 
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Specifically, the parameter estimation unit 140 calculates a reduction factor of 1/4. The matching point extraction, 

average values of the matching point vector data obtained matching point vector selection, and parameter estimation 

for the reduced image data a8 and b8 both in the horizontal for the reduced image data a2 and b2 of the reduced image 

and vertical directions and outputs the average values to the with a reduction factor of 1/2 are sequentially performed, 

matching point extraction unit 120 for using them at the s and the matching point extraction, matching point vector 

matching point extraction process of estimating the dupli- selection, and parameter estimation for the equal-size image 

cate areas of two reduced images represented by the reduced data a and b are sequentially performed, both in a hierar- 

image data a4 and b4. chical manner. 

Similarly, the matching point vector data obtained for the In the above processes, for the equal-size image data a and 

reduced image data of the reduced images with the reduction 10 b, coordinate conversion by affine transformation is per- 

f actors of 1/4 and 1/2 is also used for estimating the formed at the image conversion synthesization post-process, 

duplicate areas of two reduced images at the next matching Therefore, the matching point coordinate vectors in the 

point extraction process. horizontal and vertical directions are not necessarily in the 

As the coordinate conversion parameters for the size predetermined range so that selection of the matching point 

reduced image data a8 and b8 are output from the parameter 15 vector by the histograms is not necessary, 

estimation unit 140, the matching point extraction unit 120 The parameter estimation process will be detailed, 

extracts the matching points of the size reduced image data After the coordinate conversion parameters of the two 

a4 and b4 of the reduced images with the reduction factor of images of the image data a and b are obtained in the above 

1/4. manner, the image conversion synthesization unit 150 lastly 

The matching point extraction algorithm of the matching 20 converts the coordinates of the input image data a and b by 

point extraction unit 120 for extracting the matching points using the coordinate conversion parameters estimated by the 

of the reduced image data a4 and b4 of the reduced images parameter estimation unit 140 and synthesizes the two 

with the reduction factor of 1/4 is similar to that illustrated images into one image. 

in FIG. 10. However, used for setting the template cutting This coordinate conversion is affine transformation. In 

area at step S21 of FIG. 10 and the search area of the 25 estimating the coordinate conversion parameters from the 

matching point at step S23 are the average values both in the matching point vector of the equal-size image of the input 

horizontal and vertical directions of the matching point image data a and b, the parameter estimation unit 140 

vector coordinate data of the reduced image data a8 and b8 outputs not the average value of matching point vector data 

of the reduced images having a reduction factor of 1/8. but aflSne transformation parameters in the following man- 

FIG. 13 is a diagram illustrating a method of setting a 30 ner. 

template cutting area for the reduced image data a4. Assuming that the input image data b has a position 

In setting the template cutting area at step S21 of FIG. 10, relationship, relative to the input image data a, of 6 rotation, 
the duplicate area (indicated by oblique lines in FIG. 13) of (dx, dy) parallel motion, and m magnification, a point (xa, 
the reduced image represented by the reduced image data a4 ya) in the input image data a corresponds to a point (xb, yb) 
is estimated from the average value data (indicated by an 35 in the input image data b indicated in the following equa- 
arrow in FIG, 13) of the matching point coordinate data in tions (4). 
the horizontal and vertical directions output from the param- 
eter estimation unit 140, and the left, upper, and lower end xb = (cos0 j:a + siiid >a-ii?)x/n (4) 
positions of the template cutting area for the reduced image 

data a4 are set in accordance with the estimated duplicate 40 = A xa + B >a-t-c 

area of the reduced image of the reduced image data a4 to yb = i-iir)9'xa + cos0'ya-dy)y.m 

thereby cut the template area. 

nG. 14 is a diagram for illustrating a method of setting =-Bxa + Aya + D 
a search area of the matching point for the reduced image 

data b4. 45 

At step 823 in FIG. 10, for each of the plurality of where A^m cos 6, Wsin 9, C=-m dx, and Do-m dy. 

templates cut at step S21, a point (at the front tip of the arrow parameters A, B, C, and D are estmiated by a least 

shown in FIG. 14) is set. This point is shifted fi-om the ^^^^^ - ..^ 

template position for the image of the reduced image data a4 , '° parameter estimation umt 140 to estunate 

by the average value of the matching point coordinate data, 50 Parameters, at least two pairs of matching pomt coordi- 

in both the horizontal and vertical directions, of the reduced °f ^] °°^y P^^. matching point coordinates is 

image data a8 and b8 supplied from the parameter estima- obtained, the average value of matchmg pomt vectors of the 

tion unit 140. The search area for the matching point is set ^^^^^^^ ^^^^^^^ ^^^^ ^ reducUon 

only to the area (area S' in RG. 14) having a predetermined ^^^^^^ ^ ^ P^^^"^' "^^^i^" parameter. Namely, 

width (e.g., five pixels in the upper, lower, right, and left 55 assuming that the average value of matching point vectors of 

directions) around the set point in the upper, lower, right, and ^^^^ °f ^^e reduced image with a reduc- 

left directions factor of 1/2 is (ax, ay), then the parameters indicated 

The matchmg point extraction, matching point vector following equations (5) are used as the parallel 

selection, and parameter estimation for the reduced image motion parameters, 

data a4 and b4 of the reduced image with a reduction factor 60 * , D_n ^ 
of 1/4 are performed in the manner similar to that of the 

reduced image data a8 and b8 of the reduced image with a If no pair of matching point coordinates is obtained, the 

reduction factor of 1/8. succeeding process can not be executed. Therefore, a mes- 

. The matching point extraction process for the reduced sage to this eflfeci is displayed on the display screen to notify 

image data with a reduction factor 1/2 and that for the image 65 the operator and the synthesization process is terminated, 

data with an equal size are performed in the manner similar In this embodiment, the parameter estimation unit 140 

to that of the reduced image data of the reduced image with estimates affine transformation parameters for the equal size 
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input image, and parallel motion parameters for the reduced The synthesized image of the synthesized image data c 

images. AfEne transformation parameters may be estimated obtained in the above manner is shown in FIG. 17, An area 

for both the equal-size image and reduced images. indicated by oblique lines in FIG. 17 is not assigned with any 

The image conversion synthesization unit 150 forms pixel by the images of the image data a and b. Therefore, this 

synthesized image data c in accordance with the algorithm 5 area may be assigned dummy pixels, for example, white 

shown in FIG. 15. pixels. 

At step S51 of FIG. 15, first an image area for the In this embodiment, the image conversion synthesizatiori 

synthesized image of the synthesized image data c is set. unit 150 assigns the duplicate area with either the image data 

In this embodiment, the image area indicated by a broken a or the coordinate converted image data b. An average value 

hne in FIG. 16 is set using the coordinate system of the of the image data a and coordinate converted image data b 

image of the input image data a as a reference. Namely, the may be assigned or an average value of the image data a and 

left end of the image area corresponds to the leftmost coordinate converted image data b whose pixel values are 

coordinate value of the image of the image data a, and the weighted, for example, in the horizontal direction, may be 

right end corresponds to a larger coordinate value among assigned. 

coordinate values of the upper right and lower right corners The synthesized image data c generated by the image 

of the image of the image , data b as referenced to the conversion synthesization unit 150 is supplied to a display, 

coordinate system of the image data a. In order to convert the ^ printer, or the like. 

coordinates of the image of the image data b into the In this embodiment, image data of two gradation images 

coordinates of the image of the image data a, inverse is synthesized, pe mvention is applicable to synthesization 

transformation of the affine transformation of the equations "^^S^ ^^^^ color unages. In this case, for example 

(4) is used. Namely, assuming the inverse transformation extractmg matching pointe of two images, only image of 

parameters are A, B',C, and D', transformation indicated by ^ components may be used among the color image com- 

ihe following equations (6) is performed. ponents of R (red), G (green) and B (blue), or an u^age 

* ^ \ / r formed by average values of R, G, and B may be used. 

xa^'xb^B'yb+c lu embodiment, image data of two images taken with 

yo'-B'-xlH-A'yb+D' (6) 25 an electronic still camera, a video camera, or the like is 

synthesized. Obviously, the invention is applicable to syn- 

where A'«A/(A^+B^), B'«-B/(A^+B^) C'«(-A-C+B D)/ thesization of computer graphics image data. 

(A^+B^), and D'«(-B C-A D)/(A^+B^). As described above, according to the image synthesizing 

The upper end of the synthesized image of the synthesized apparatus of this embodiment, a plurality of images can be 

image data c corresponds to a coordinate value of the upper 30 precisely synthesized without cumbersome operations such 

end of the image of the image data a and a smaller coordinate as designating a matching point in a plurality of images with 

value among coordinate values of the upper right and upper a pointer or the like. 

left corners of the image of the image data b as referenced In the image synthesizing apparatus of this embodiment, 

to the coordinate system of the image data a. The lower end the template cutting area, from which small area templates 

of the synthesized image of the synthesized image data c 35 are cut to obtain a matching point in each image of a 

corresponds to a coordinate value of the lower end of the plurality of images, is designated in a limited area, 

image of the image data a and a larger coordinate value Therefore, erroneous matching point extraction (by identi- 

among coordinate values of the lower right and lower left fying a non-existent matching point in an area other than the 

corners of the image of the image data b as referenced to the duplicate areas of a plurality of images is avoided and a 

coordinate system of the image data a. 40 synthesized image can be formed precisely. 

Next, at step S52 of FIG. 15, a connection position is set In the image synthesizing apparatus of this embodiment, 

to the center of the duplicate areas as indicated by a dotted the search area, for searching a matching point in each image 

line in FIG, 16. The connection position is indicated by an corresponding to the template cutting area from which small 

average of the right end coordinate value of the image of the area templates are cut to obtain the matching point in each 

image data a and a smaller coordinate value among the 45 image of a plurality of images, is designated in a limited area 

coordinate values of the upper left and lower left corners of with reference to the information supplied with the param- 

the image of the image data b as referenced to the coordinate eter estimation unit. Therefore, a plurality of images can be 

system of the image data a. synthesized in a short process time. Even if a plurality of 

At step S53 of FIG. 15, pixel values in the image area of images contains a number of analogous parts, the matching 

the synthesized image data c set at step S51 are obtained and 50 point can be precisely extracted. A plurality of images can 

written in a memory area. therefore be synthesized precisely. 

At step S54 of FIG, 15, for the image area of the image In the image synthesizing apparatus of this embodiment, 

data a in the image area of the synthesized image data c, the the search area, for searching a matching point in each image 

pixel values of the image data a themselves are written in a corresponding to the template cutting area from which small 

memory area corresponding to the image data a in the 55 area templates are cut to obtain the matching point in each 

memory area corresponding to the image area of the syn- image of a plurality of images, is designated in a limited area 

thesized image data c. with reference to the extraction results of the matching point 

At step S55 of FIG, 15, for the image area of the of a reduced image. Therefore, a plurality of images can be 

coordinate converted image data b in the image area of the synthesized in a short process time. Even if a plurality of 

synthesized image data c, the pixel values of the coordinate 60 images contains a number of analogous parts, the matching 

converted image data b themselves are written in a memory point can be precisely extracted. A plurality of images can 

area corresponding to the coordinate converted image data b therefore be synthesized precisely, 

in the memory area corresponding to the image area of the In the image synthesizing apparatus of this embodiment, 

synthesized image data c. In this case, if the pixel values of many matching points are extracted near at the connection 

the image data a have already been written, only the pixels 65 position of image synthesization. Inhere fore, a shift of the 

to the right of the connection position are written. Step S56 connection position of the synthesized image can be 

is similar to S26. reduced. 
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FIG. 18 shows the structure of an image synthesiziDg 
apparatus of the fourth embodiment of the invention. In FIG. 
18, like elements having functions similar to those of the 
third embodiment shown in FIG. 8 are represented by using 
identical reference numerals and the detailed description 5 
thereof is omitted. 

In FIG. 18, reference numerals 113 and 114 represent a 
feature extraction unit which emphasizes the edge portion of 
a reduced image of the reduced image data output from the 
image reduction unit 111, 112 to thereby generate feature lO 
image data which is stored in an image memory of the 
feature extraction unit 113, 114. The feature image data is 
represented hereinafter by a2', b2', a4', b4', a8*, and b8'. 

In the image synthesizing apparatus of the fourth 
embodiment, an image memory is provided not in the image 35 
reduction units 111 and 112, but in the next stage feature 
extraction units 113 and 114, The generated feature image 
data is stored in this memory. In this embodiment, in order 
to generate the feature image data, the reduced image data 
is subjected to convolution operation by a 3x3 pixel two- 20 
dimensional filter shown in FIGS. 9 A and 9B to emphasize 
the edge portion of the reduced image of the reduced image 
data. 

Reference numeral 121 represents a matching point 
extraction unit which extracts the matching points of the 25 
images represented by the image data a and b supplied from 
the input terminals lOOfl and 100/? or by the feature image 
data generated by the feature extraction units 113 and 114. 

The feature extraction unit 121 extracts matching points 
of the feature image data a8' and b8' of the feature images 30 
having a size of 1/8 the images of the input image data a and 
b, and then extracts the matching points of the feature image 
data having a reduction factor of 1/4, 1/2, an equal size, and 
the like. In extracting the matching points of the feature 
image data having a reduction factor of 1/4, 1/2, an equal 35 
size, and the like, estimation at the duplicate area is per- 
formed by using the conversion parameters formed by the 
matching points extracted previously. 

The matching point extraction algorithm to be executed 
by the matching point extraction unit 121 will be described 40 
with reference to the flow chart shown in FIG. 20. 

Referring to FIG. 20, at step S31 a template cutting area 
for the feature image data a8' is set. In extracting the 
matching points of the feature image data a8' and b8', since 
the duplicate areas of the two feature images cannot be 45 
known in advance, a predetermined area is set as the 
template cutting area similar to the third embodiment. 

The following steps S33 to S3 6 are executed for all the 
templates cut at step S33 of FIG. 20 (step S37 is similar to 
S26). 50 

At step S33, templates are cut around the features. 

Specifically, in extracting the matching point of the fea- 
ture image data a8' with a reduction factor of 1/8, as shown 
in FIG. 21, the template cutting area set at step 831 is 
divided into the upper and lower areas Tl and T2. The 55 
matching points in the areas Tl and T2 are extracted at steps 

533 to S36 along arrows shown in FIG. 20, i.e., from the 
right end of the feature image data a8' until a predetermined 
number of matching points are extracted. 

Extraction of the matching point is judged by whether the 60 
pixel value of the feature image of the feature image data a8* 
is a predetermined threshold value or larger. If it is a 
predetermined threshold value or larger, it is judged as the 
feature image and the matching point is extracted at steps 

534 to S36 of FIG. 20, If the pixel value is not the 65 
predetermined threshold value or larger, it is judged the 
image is not the feamre image and the matching point is not 
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extracted at steps S34 to S36 of FIG. 20, and the feature 
image process for the next image size is performed. 

In this embodiment, the process continues until the pre- 
determined number of matching points are extracted and 
then the process is terminated. Therefore, unnecessary pro- 
cess time can be saved and the process time can be short- 
ened. Since the process is performed from the right end of 
the feature image data a8', many matching points can be 
extracted near the connection position of two images, for 
example, even if the process is terminated sooner than would 
conventionally happen. Therefore, a shift of the connection 
position of two images can be reduced. Since the template 
cutting area is divided into the upper and lower areas, the 
connection position can be balanced between the two 
images. 

Similar processes are performed for extracting the match- 
ing point of the feature image data having a reduction size 
of 1/4 and 1/2. In extracting the matching point of the feature 
image data of an equal size, the next stage image conversion 
synthesization unit 150 sets the connection position of two 
images as shown in FIG. 16 similar to the third embodiment. 
Therefore, the matching point is extracted near at the con- 
nection position. The connection position of two images is 
calculated in the manner similar to step S52 of FIG. 15 to be 
executed by the image conversion synthesization unit 150 of 
the third embodiment. 

After the connection position of two images is calculated, 
the template cut area set at step S31 is divided into right and 
left areas T3 and T4 at the boundary of the connection 
position as shown in FIG, 22. The process is executed along 
arrows shown in FIG. 22, i.e., starting at the position near the 
connection position (L' in FIG, 22) of the feature image data 
a. 

The template cutting area is set in the similar manner for 
extracting the matching point of the feature image data a2' 
and a4' having a reduction factor of 1/4 and 1/2. For, 
extracting the matching point of the equal -size feature image 
data a, the convolution operation is performed by using a 
3x3 pixel two-dimensional filter shown in FIG. 19 and 
thereafter the matching point is extracted through a thresh- 
old process to cut templates around the feature, because the 
process by the feature extraction units 113 and 114 is not 
performed for the equal-size image data a. 

Next, at step S34 of FIG, 20, a search area for the right 
side image data b is set for searching the matching point for 
the template cut at step S33. The duplicate areas are not 
known in advance for the feature image a8' and b8' having 
a reduction factor of 1/8. Therefore, similar to the third 
embodiment, a predetermined area is set as the search area. 
For the feature data of the feature images having a reduction 
factor of 1/4, 1/2, and the equal size, and the like, the search 
area is set by using coordinate conversion parameters output 
from the parameter estimation unit 140 similar to the third 
embodiment. 

At step S35 of FIG. 20, the image in the search area for 
searching the matching point of the right side feature image 
data b8' is compared with the image in the template for the 
left side feature data a8', while sequentially shifting the 
template in parallel. A difference value between pixel values 
of the image in the template and the image in the search area. 
The position of the template where the sum of absolute 
values of the difference values is minimum, is used as the 
matching point. 

At step S36 of FIG. 20, reliability of the matching point 
detected at step S35 is checked. Coordinate data of the 
extracted matching point is stored in the intemal memory of 
the matching point extraction unit 121. 
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In this embodiment, the matching point of the reduced 
image is extracted from the feature image. Therefore, even 
if there is some luminance difference between input images, 
the matching point can be extracted without being affected 
by the luminance difference. Since the matching point is 
extracted from the input image itself without using the 
feature image, the matching point can be extracted without 
emphasizing image noises. 

The other operations of the image synthesizing apparatus 
of the fourth embodiment are similar to the third 
embodiment, and so the description thereof is omitted. 

FIG. 23 shows the structure of an image synthesizing 
apparatus according to the fifth embodiment of the inven- 
tion. In FIG. 23, like elemenLs having functions similar to 
those of the third embodiment shown in FIG. 8 are repre- 
sented by using identical reference numerals and the detailed 
description thereof is omitted. 

In this embodiment, the matching point extraction unit 
220 extracts the matching point by using the matching point 
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cutting area from which small area templates are cut to 
obtain the matching point in each image of a plurality of 
images, is designated in a limited area with reference to the 
information output from the matching point vector selection 
unit. Therefore, a plurality of images can be synthesized in 
a short process time. Even if a pliu*ality of images contains 
a number of analogous parts, the matching point can be 
precisely extracted. A plurality of images can therefore be 
synthesized precisely. 

As described above, according to the image synthesizing 
apparatus of this embodiment, even if there is some bright- 
ness difference between input images, the matching point 
can be extracted in a short process time without cumbersome 
operations. A plurality of images can be automatically and 
precisely synthesized with a small shift of the connection 
position between images. 

Many widely different embodiments of the present inven- 
tion may be constmcted without departing from the spirit 
and scope of the present invention. It should be understood 



coordinate data output from the matching point vector 20 that the present invention is not limited to the specific 

selection unit 130. embodiments described in the specification except as 

In this embodiment, the matching point extraction algo- defined in the appended claims, 

rithm to be executed by the matching point extraction unit What is claimed is: 

220 is similar to the third embodiment shown in FIG. 10. 1. An image synthesizing method for synthesizing a 

The matching point extraction algorithm to be executed by 25 plurality of images, comprising the steps of: 



the matching point extraction unit 22 will be described with 
reference to the flow chart shown in FIG. 10. 

At step S21 shown in FIG. 10, a template cutting area for 
the feature image data a8 is set. The following steps S23 to 
S25 are executed for all the templates cut at step S22. The 30 
matching points of the reduced images having a reduction 
factor of 1/4, 1/2, 1/1 and the like are extracted by using only 
the templates which can extract the matching points of the 
size reduced image data of reduced images having a reduc- 
tion size of 1/8, 1/4, 1/2 and the like. 35 

At step S23 of FIG. 10, for each of the templates cut in the 
above manner, a search area for the right side image data b8 
is set for searching the matching point. The duplicate areas 
are not known in advance for the reduced image data a8 and 
b8. Therefore, similar to the third embodiment, a predeter- 40 
mined area is set as the search area! In extracting the 
matching point of the size reduced image data having a 
reduction factor of 1/4, 1/2, the equal size, the search area is 
set by using matching point coordinate data output from the 
matching point vector selection unit 30, this search area 45 
being an area having a predetermined with (e.g., +/-5 pixels) 
in the upper, lower, right, and left directions around the 
matching point coordinate value. 

At step S24 of FIG. 10, the image in the search area for 
searching the matching point of the right side feature image 50 
data b8 is compared with the image in the template for the 
left side feature data a8, while sequentially shifting the 
template, A difference value of pixel values between the 
image in the template and the image in the search area is 
calculated. The position of the template where the sum of 55 
absolute values of the difference values is minimum, is used 
as the matching point. 

At step S25 of FIG. 10, reliability of the matching point 
detected at step S24 is checked. Coordinate data of the 
extracted matching point is stored in the internal memory of 60 
the matching point extraction unit 121. 

The other operations of the image synthesizing apparatus 
of the fifth embodiment are similar to the third embodiment, 
and so the description thereof is omitted. 

As described above, in the image synthesizing apparatus 65 
of this embodiment, the search area, for searching a match- 
ing point in each image corresponding to the template 



(a) inputting the plurality of images; 

(b) detecting those ones of the images which at least 
partially overlap with another selected image; 

(c) detecting those ones of the images which at least 
partially overlap with at least of said detection images, 
and which have not already been detected or selected; 

(d) detecting position relationships between a predeter- 
mined reference image and those ones of images which 
have been detected or selected in said steps (b) and (c); 
and 

(e) collectively syntheszing all of the images on the basis 
of the position relationships detected in said step (d). 

2. An image synthesizing method according to claim 1, 
wherein said synthesizing step obtains image data of an 
image in a synthesized image area from the input images in 
accordance with the position relationship between the pre- 
determined reference image and an image which has been 
detected in said steps (b) and (c). 

3. An image synthesizing method for synthesizing a 
plurality of images, comprising the steps of: 

(a) inputting the plurality of images; 

(b) generating a process list by detecting those ones of the 
images which at least partially overlap with another, 
selected one of the images, and, for at least one of the 
detected images, by detecting those one of the images 
which at least partially overlap with said detected 
images, and which have not already been detected or 
selected; 

(c) detecting position relationship betwen a predeter- 
mined reference image and those ones of images which 
have been detected or selected in said step (b); 

(d) determining conversion parameters based on the posi- 
tion relationships detected in said step (c); and 

(e) synthesizing all of the images on the basis of the 
conversion parameters determined in said step (d). 

4. An image synthesizing method for synthesizing a 
plurality of images, comprising the steps of: 

producing reduced versions of the plurality of images; 
applying the reduced versions of the plurality of images to 
at least a first input of an extractor which extracts 
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similar points in the reduced versions of the plurality of 
images and predicts duplicated portions of the plurality 
of images based on information applied to a second 
input of the extractor; 

generating coordinate conversion parameters based on the 
similar points extracted by the extractor, and applying 
information representing the coordinate conversion 
parameters to the second input of the extractor; and 

synthesizing the plurality of images baBed on the infor- 
mation representing the coordinate conversion param- 
eters generated in the generating step. 

5. A method according to claim 3, wherein the selected 
image Ls disposed substantially at a center of the plurality of 
images, 

6. An image synthesizing method for synthesizing a 
plurality of images, comprising the steps of: 

(a) inputting the plurality of images; 

(b) generating a process list by detecting those ones of the 
images which at least partially overlap with another, 
selected one of the images, and, for at least one of the 
detected images, by detecting those one of the images 
which at least partially overlap with said detected 
image, and which have not already been detected or 
selected; 

(c) detecting position relationships between said images 
which have been detected or selected in said step (b); 

(d) determining conversion parameters based on the posi- 
tion relationships detected in said step (c); 

(e) selecting a reference image; 

(f) converting said conversion parameters determined in 
said step (d) with respect to the reference image 
selected in said step (e); and 

(g) synthesizing all of the images on the basis of the 
conversion parameters converted in said step (f). 

7. An image synthesizer for synthesizing a plurality of 
images, comprising: 

means for imputting the plurality of images; 

means for detecting those ones of the image which at least 
partially overlap with a selected one of the images, and, 
for at least one of the detected images, detecting those 
ones of the images which at least partially overlap with 
that at least one detected image, and which have not 
already been detected or selected; 

means for detecting position relationships between each 
selected image and those ones of images which have 
been detected as being at least partially overlapping 
with the selected image; and 

means for collectively synthesizing all of the images on 
the basis of the detected position relationships. 

8. An image synthesizer according to claim 7, wherein 
said means for collectively synthesizing obtains image data 
of an image in a synthesized image area from the input 
images in accordance with the position relationship between 
each selected image and an image which has been detected 
as bein at least partially overlapping with that selected 
image. 

9. A image synthesizer for synthesizing a plurality of 
images, comprising: 

means for inputting the plurality of images; 

menas for generating a process list by detecting those 
ones of the images which at least partially overlap with 
a selected one of the images, and, for at least one of the 
delected images, by detecting those one of the images 
which at least partially overlap with that at least one 
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detected image, and which have not already been 

detected or selected; 
menas for detecting position relationships between each 

selected image and those one of images which have 

been detected as being at least partially overlapping 

with that selected image; 
means for determining conversion parameters base on the 

position relationships detected; and 
means for synthesizing all of the images on the basis of 

the determined conversion parameters. 

10. A method according to claim 9, wherein the selected 
image is disposed substantially at a center of the plurality of 
images. 

11. An image synthesizer for synthesizing a plurality of 
images, comprising: 

means for producing reduced versions of the plurality of 
images; 

means for applying the reduced version of the plurality of 
images to at least a first input of an extractor which 
extracts similar points in the reduced versions of the 
plurality of images and predicts duplicated portions of 
the plurality of images based on information applied to 
a second input of the extractor; 

means for generating coordinate conversion parameters 
based on the similar points extracted by the extractor, 
and applying information representing the coordinate 
conversion paramters to the second input of the extrac- 
tor; and 

means for synthesizing the plurality of images based on 
the information representing the generated coordinate 
conversion parameters. 

12. An image synthesizer for synthesizing a plurality of 
images, comprising: 

means for inputting the plurality of images; 

means for generating a process list by detecting those 
ones of the images which at least partially overlap with 
another, selected one of the images, and, for at least one 
of the detected images, by detecting those one of the 
images which at least partially overlap with that at least 
one detected image, and which have not already been 
detected or selected; 

means for detecting position relationships between each 
selected image and those one of images which have 
been detected as being at least partially overlapping 
with that selected image; 

means for determining conversion parameters based on 
the detected position relationships; 

means for synthesizing all of the images on the basis of 
the determined conversion parameters; 

means for selecting a reference image; and 

means for converting at least one of the parameters with 
respect to the selected reference image. 

13. An image synthesizing apparatus for synthesizing a 
plurality of images, comprising: 

an input unit for inputting the plurality of images; 

a first detection unit for detecting those ones of the images 

which at least partially overlap with another selected 

image; 

a second detection unit for detecting those ones of the 
images which at least partially overlap with at least one 
of said detected images detected by first detection unit, 
and which have not already been detected or selected; 

a position relationship detection unit for detecting posi- 
tion relationships between a predetermined reference 
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image and those ones of iniages which have been 
detected or selected by said first and second detection 
unit; and 

a synthesizing unit for collectively synthesizing all of the 
images on the basis of the position relationships 
detected by said position relationship detected unit. 

14. An apparatus according to claim 13, wherein said 
synthesizing unit obtains image data of an image in a 
synthesized image area from the input images in according 
with the position relationship between the predetermined 
reference image and an image which has been detected by 
said first and second detection unit. 

15. An image synthesizing apparatus for synthesizing a 
plurality of images, comprising: 

an input unit for inputting the plurality of images; 

a generating unit for generating a process list by detecting 
those ones of the images which at least partially overlap 
with another selected image, and, for at least one of the 
detected images, by detecting those ones of the images 
which at least parially overlap with said detected 
images, and which have not already been detected or 
selected; 

a detecting unit for detecting position relationships 

between a predetermined reference image and those 

ones of images which have been detected or selected by 

said generating unit; 
a determining unit for determining conversion parameters 

base on the position relationships detected by said 

detecting unit; and 
a synthesizing unit for synthesizing all of the images on 

the basis of the conversion parameters determined by 

said determining unit. . 

16. An apparatus according to claim 15, wherein the 
selected image is disposed substantially at a center of the 
plurality of images. 

17. An image synthesizing apparatus for synthesizing a 
plurality of images, comprising: 

a producing unit for producing reduced versions of the 

plurality of images; 
an applying unit for applying the reduced versions of the 

plurality of images to at least a first input of an extractor 
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which extracts similar points in the reduced version of 
the plurality of images and predicts duplicated portions 
of the plurality of images based on information applied 
ot a second input of the extractor; 
^ a generating unit for generating coordinate conversion 
parameters based on the similar points by the extractor, 
and applying information representing the coordinate 
conversion parameters to the second input of the 
extractor; and 

a synthesizing unit for synthesizing the plurality of 
images based on the information representing the coor- 
dinate conversion parameters generated by said gener- 
ating unit. 

15 18. An image synthesizing apparatus for synthesizing a 
plurality of images, comprising: 

an input for inputting the plurality of images; 
a generating unit for generating a process list by delecting 
2Q those ones of the images which at least partially overlap 
with another selected image, and, for at least one 
detected image, by detecting those ones of the images 
which at least partially overlap with said detected 
images, and which have not already been detected or 
25 selected; 

a detecting unit for detecting position relationships 
between said images which have been detected or 
selected by said generating unit; 
a determining unit for determining conversion parameters 
based on the position relationships delected by said 
detecting unit; 
a selecting unit for selecting a reference image; 
a converting unit for converting at least one of said 
35 parameters determined by said determining unit with 
respect to the reference image selected by said selecting 
unit; and 

a synthesizing unit for synthesizing all of the images on 
the basis of the conversion parameters converted by 
said converting unit. 

♦ « * * * 
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